Systems and methods for remote presence

ABSTRACT

Remote presence systems and methods are presented. In one embodiment, a system may comprise a pilot workstation comprising a pilot computing station having a display, a microphone, a camera oriented to capture images of the pilot, a network connectivity subsystem, and a master input device such as a keyboard, mouse, or joystick. The pilot network connectivity subsystem may be operatively coupled to an electromechanically mobile workstation comprising a mobile base interconnected to a head component. The mobile workstation may comprise a display, a microphone, a camera oriented to capture images of nearby people and structures, and a workstation network connectivity subsystem that preferably is operatively coupled to the pilot network connectivity subsystem. Preferably by virtue of the system components, the pilot is able to remotely project a virtual presence of himself in the form of images, sound, and motion of the mobile workstation at the location of the mobile workstation.

RELATED APPLICATION DATA

The present application is a continuation application of U.S. patentapplication Ser. No. 15/850,679, filed on Dec. 21, 2017, which is acontinuation of U.S. patent application Ser. No. 15/391,597, filed onDec. 27, 2016 now abandoned, which is a continuation application of U.S.patent application Ser. No. 15/006,036, filed on Jan. 25, 2016 nowabandoned, which is a continuation application of U.S. patentapplication Ser. No. 14/035,223 filed on Sep. 24, 2013 now abandoned,which claims the benefit under 35 U.S.C. § 119 to U.S. ProvisionalApplication Ser. No. 61/744,378 filed Sep. 24, 2012 and U.S. ProvisionalApplication Ser. No. 61/705,625 filed Sep. 25, 2012. The foregoingapplications are hereby incorporated by reference into the presentapplication in their entirety.

FIELD OF THE INVENTION

The present invention relates generally to telecommunication systems,and more particularly to remote presence systems wherein aremotely-positioned pilot may utilize an electromechanically mobileworkstation local to another work environment to virtually participatein that environment.

BACKGROUND

The notion of telepresence via the use of remotely locatedelectromechanical systems, such as the CanadaArm utilized on the secondlaunch of the NASA Space Shuttle in 1981, has been around for decades,and now plays a role in many marketplaces, from shopping to surgery.Further, with the onset of video conferencing in many forms, fromhigh-end and high-expense systems like those available from thetelepresence division of Cisco Systems, Inc., to the nearly free systemsavailable on almost any personal computer by the Skype division ofMicrosoft Corporation, people are becoming more and more interested inutilizing video conferencing techniques to communicate and shareinformation. More recently, a marketplace known as “remote presence” hasevolved, wherein certain aspects of electromechanical telepresence andvideo conferencing may be combined. A typical remote presence systemwill involve a pilot workstation, such as a laptop having specifichardware, software, and network connectivity, positioned in a firstlocation usually at least somewhat remote from a second location whereinan electromechanically movable portion of the remote presence system isconfigured to assist the pilot in being “virtually present” in thesecond location. Typically this involves facilitating remote navigationof the system local to the second location from the first location whilealso facilitating real-time or near real-time video conferencing throughthe remote presence system. Several systems have entered themarketplace, but have had limited success due to various factors,including their complexity, expense, focus on two-wheeled balancing,lack of network connectivity robustness, operational noise level, and/orsuboptimal user interface configurations. There is a need for aneffective and affordable remote presence system that can be used andrelied upon for modern business demands.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B illustrate various aspects of embodiments of a remotepresence system in accordance with the present invention.

FIGS. 2A-2B illustrate user interface views pertinent to embodiments ofa remote presence system in accordance with the present invention.

FIGS. 2C-2H illustrate system operational views pertinent embodiments ofto a remote presence system in accordance with the present invention.

FIGS. 2I-2J illustrate orthogonal views pertinent to embodiments of afully-assembled remote mobile workstation of remote presence system inaccordance with the present invention.

FIGS. 2K-2S illustrate orthogonal views pertinent to embodiments of apartially-assembled remote mobile workstation of remote presence systemin accordance with the present invention, with emphasis on mobile basecomponentry.

FIGS. 2T to 2Z-12 illustrate orthogonal views pertinent to embodimentsof a partially-assembled remote mobile workstation of remote presencesystem in accordance with the present invention, with emphasis on headcomponentry.

FIGS. 3A-3I illustrate schematics and charts pertinent to embodiments ofan audio subsystem for a remote presence system in accordance with thepresent invention.

FIG. 4 illustrates a schematic pertinent to an embodiment of an accessgateway in accordance with the present invention.

FIGS. 5A-5D illustrate orthogonal views pertinent to embodiments ofcamera subsystems for a remote presence system in accordance with thepresent invention.

FIGS. 6A-6G illustrate views pertinent to embodiments of a pilotworkstation and user interface for a remote presence system inaccordance with the present invention.

FIGS. 7A-7H illustrate views pertinent to embodiments of a remote mobileworkstation and charging dock for a remote presence system in accordancewith the present invention.

FIGS. 8A-8C illustrate schematics pertinent to embodiments of a drivemotor configuration for a remote presence system in accordance with thepresent invention.

FIG. 9 illustrates a typical office environment map.

FIG. 10 illustrates one embodiment of a connectivity configurationbetween a remote mobile workstation and a pilot workstation inaccordance with the present invention.

FIG. 11 illustrates one embodiment of a connectivity configurationbetween a remote mobile workstation, a pilot workstation, and apassenger bridge in accordance with the present invention.

SUMMARY OF THE INVENTION

Remote presence systems and methods are presented. In one embodiment, asystem may comprise a pilot workstation comprising a pilot computingstation having a display, a microphone, a camera oriented to captureimages of the pilot, a network connectivity subsystem, and a masterinput device such as a keyboard, mouse, or joystick. The pilot networkconnectivity subsystem may be operatively coupled to anelectromechanically mobile workstation comprising a mobile baseinterconnected to a head component. The mobile workstation may comprisea display, a microphone, a camera oriented to capture images of nearbypeople and structures, and a workstation network connectivity subsystemthat preferably is operatively coupled to the pilot network connectivitysubsystem. Preferably by virtue of the system components, the pilot isable to remotely project a virtual presence of himself in the form ofimages, sound, and motion of the mobile workstation at the location ofthe mobile workstation.

One embodiment is directed to a remote presence system, comprising aremote mobile workstation comprising a controller and a drive system;and a pilot workstation operatively coupled to the remote mobileworkstation in a pier-to-pier network configuration such that commandsfrom a pilot operating a first master input device comprising the pilotworkstation are communicated to the remote mobile workstation andutilized to navigate the drive system with commands from the controller;wherein the controller of the remote mobile workstation is configuredsuch that navigation commands from the first master input device will beignored if navigation commands from a second master input devicecomprising the pilot workstation are instituted.

DETAILED DESCRIPTION

Referring to FIG. 1A, in one embodiment, a remote presence systemcomprises a pilot computing system (10) for a pilot (8) who may belocated at a location remote from the location of a remote mobileworkstation (4). For simplicity, these locations may be referred to asthe “pilot location” and the “remote location”; persons physicallylocated at the remote location may be referred to as “locals”. Thesystem also may comprise a charging base or charging dock (2) located atthe remote location and configured to re-charge a battery component ofthe remote mobile workstation (4) when the remote mobile workstation isparked upon the charging dock (2). In one embodiment, the pilot (8) mayremotely operate the remote mobile workstation via his pilot computingsystem (10) using conditional network connectivity (1), which isconditioned upon the pilot, at least initially, providing authenticationinformation through a login gateway, or “lobby” (6), which generally isoperated by a server remote to both the remote mobile workstation andpilot. The pilot computing system (10) may comprise a laptop computer,desktop computer, or smaller or larger computing system that may beoperated at the pilot location by the pilot, such as a smartphone orother device, so long as such device has the basic communicationshardware and software required for operating the remote mobileworkstation remotely, as discussed below. In another embodiment, a pilot(8) workstation may be configured to authenticate directly to the remotemobile workstation (4) without involving a central authenticationservice. For example, in one configuration a pilot could use a direct IPaddress to connect directly to a selected mobile workstation (4). FIG.1B illustrates a series of charging docks (2) located in a particularcorporate facility, two of which are occupied by charging remote mobileworkstation (4) systems.

Referring to FIG. 2A, in operation, a lobby log-in screen (12)facilitates access to a remote mobile workstation selection controlpanel interface (14), whereby the pilot may select a particular remotemobile workstation that he wishes to operate from his location using hispilot computing system. As shown in the exemplary interface view (14),one particular location (such as Seattle, listed in the interface view)may have numerous remote mobile workstation systems available, similarto the scenario depicted in FIG. 1B. Referring to FIG. 2C, uponselecting a particular remote mobile workstation (4), video images ofthe pilot (14) acquired from the pilot computing system (10) aredelivered to the display of the remote mobile workstation (4) and theremote mobile workstation (4) becomes actively operated by the pilotfrom his pilot location. A downward-facing light (18) assists withlighting the field of view of a navigation camera, discussed furtherbelow, when the remote mobile workstation (4) is actively being operatedby a pilot. The system may also comprise one or more forward-facinglights or sensors, such as one or more infrared sensors, laserrangefinding sensors, and/or sonar detectors. Further, various colors oflaser pointer tools may be coupled to the mobile workstation (4) toassist with navigation and pointing by a remote pilot. The pilot userinterface may be configured to not only allow pointing with a laserdevice, but also projection with a projector, such as a laser projectoror conventional light projector which may be operatively coupled to theremote workstation (4). Projectors may be utilized to not only assistwith pilot communication, but also to display various augmented realityobjects or to display photos, artwork, and/or advertisements for peoplelocal to the mobile workstation (4). Infrared imaging using one or moreinfrared cameras may be utilized to provide safety to people and objectslocal to the mobile workstation (4) by providing enhanced vision ofobjects and personnel, such as in the dark, to prevent collisions, toprevent movement of the workstation (4) into too-close a proximity witha cliff or stairwell, and the like. Referring to FIG. 2D, the pilot hasdriven the remote mobile workstation (4) out of the charging dock (2)and into the floor space ahead.

Referring to FIG. 2E, the pilot is shown operating the remote mobileworkstation (4) in a corporate meeting setting near a conference table(28) and three locals (30) who are engaging the pilot as if she was inthe room. Images of the locals are captured by a main camera (20), whilethe pilot's voice is projected through a front-oriented speaker (24). Inone embodiment the speaker (24) may comprise a single sound driver. Inanother embodiment, the speaker (24) may comprise a beam forming,parametric array, or stereo speaker, which may be configured to projectsound directionally to specific areas, under the direction of the pilotthrough the pilot user interface. In one embodiment, the speaker (24),or entire head component (36) may be movably coupled to the remainder ofthe remote mobile workstation (4) to assist with greater field of viewfor camera systems, greater perception of presence of the pilot in thelocal environment, and greater speaker directionalization capability.For example, in one embodiment the head (36) may be movably coupled tothe other associated components of the workstation (4), such as thespine members (26), with a movable interface that allows forpilot-controllable degrees of freedom for head pan, head tilt, and/orhead elevation relative to the spine members (26). In another embodimentthe speaker may be movably coupled to the head (36) with speaker pan,speaker tilt, and/or speaker elevation degrees of freedom which may beat least partially controlled by the pilot through the pilot userinterface. The spine members of the remote mobile workstation (4) areselected in the depicted embodiment to provide desired conversationheight (to approximately match the height of the faces of thecounterparties to the particular conversation) in both a conference roomsetting, as in FIG. 2E, and in a standing type of setting, as in FIG.2F, wherein a local (30) is engaging the pilot in a hallway.

Referring to FIGS. 2G and 2H, the pilot is shown navigating the remotemobile workstation (4) back to a charging dock (2) where it may beparked for charging and use by others. A downward-facing navigationcamera (22) provides a downward-oriented field of view (32) that assistswith navigation of the remote mobile workstation (4) around variousobjects on the floor, such as portions of the charging dock (2). In oneembodiment, the main camera (20), the navigation camera (22), or bothcameras (20, 22) may be a wide angle camera, which may capture a fieldof view or field of capture between about 100 degrees and about 140degrees.

Referring to FIG. 2I, three main components of one embodiment of aremote mobile workstation (4) comprise a mobile base (34), a headassembly (36), and a spine assembly (26) interconnecting the head (36)and base (34). FIG. 2J shows a different orthogonal view of the samesystem depicted in FIG. 2I.

FIGS. 2K-2S illustrate various aspects of an embodiment of a mobile baseassembly (34) in various states of disassembly.

Referring to FIGS. 2K and 2L, the mobile base assembly comprises afive-wheeled device driven by two main wheels in a differential driveformation, with the two main drive wheels (40) opposed directly acrossfrom each other. With the upper skin component removed, the innerworkings and parts of the mobile base (34) may be seen. Spine-mountingstems (38) are coupled to the mobile base (34) for interconnection withthe spine assembly (26) tubular structures shown in FIGS. 2I-2J. Element42 is a thin tire element which may be selected to provide greater floorgripping and noise and vibration dampening, such as a polymer such aspolyurethane. A relatively large battery cover (54) reveals that asignificant amount of the volume of the mobile base (34) is occupied byan underlying battery. The mobile base (34) is surrounded by impactresistant bumpers (44, 46) configured to assist in preventing damage tothe mobile base componentry upon impact with a wall or other structure.A base motherboard (48) comprises motor control components as well as agyro component configured to sense forward pitching rotation as well asroll rotation (i.e., about an axis perpendicular from the floor throughthe center of the mobile base 34), and a 3-axis accelerometer to detectacceleration in X, Y, and Z Cartesian directions relative to the mobilebase (34). In one embodiment a 3-axis accelerometer and 3-axis gyro maybe utilized to assist as inputs for smoothing out motion control of themobile system (4).

Referring to FIG. 2M, a bottom view shows the front wheel caster (50)movably coupled to the front wheel suspension arm (52), which isrotatably coupled to the main frame of the mobile base (34) as discussedbelow. The two main wheels (40) in differential drive configuration areshown opposite each other, as are the rear caster wheels (56). One ormore charging contacts (58) configured to directly interface with matingfeatures on a charging dock are featured on the underside of the mobilebase (34), and may be electroplated with a high-conduction material suchas gold to facilitate quality charging conduction. In one embodiment,each of the main wheels (40) may be operatively coupled to a rotationalposition encoder configured to assist in monitoring the rotationalposition of each wheel with a high degree of accuracy, which may beutilized to provide positional information regarding the wheels andmobile workstation (4) in general for the remote pilot and controlsystem. In one embodiment, an automatic docking feature may allow for apilot to identify, or confirm semiautomatic or automatic identificationof, a charging dock (2), after which the mobile workstation may beconfigured to automatically navigate onto the charging dock (2) and intoa power charging position. The cameras (20, 22) or other sensors, suchas Bluetooth™ sensors, near field communication sensors, or varioustypes of wireless beacons which may be coupled to the remote workstation(4) and/or the charging dock (2) may be utilized in the identificationof and/or navigation to such an auto-docking coupling. In embodimentswherein any kind of automatic navigation is to be conducted, the systemmay be configured to require affirmative input from the pilot duringsuch automatic navigation or the navigation will halt. For example, inone embodiment, when a desired charging dock (2) has been identified andconfirmed by a pilot, the system may be configured to require that thepilot press a certain user interface button or key constantly (i.e.,hold such key or button down), or in another embodiment continued cyclicdepression and release of such button or key at a frequency at least asgreat as a predetermined baseline frequency, to maintain the automaticnavigation movement while the mobile workstation (4) navigates to thecharging dock (2) and parks. These so-called “live-man” approaches mayassist in preventing any accidental automatic navigation issues. Similarlive-man functionality may be utilized to prevent unexpected oraccidental inputs using various master input devices. For example, if apilot workstation has a keyboard as well as a joystick for master inputdevices to a connected mobile workstation (4), the system may beconfigured such that inputs to the joystick will only be interpreted asdrive commands if a particular live-man key or button is also depressedin a manner as describe above, to prevent accidental inputs from thejoystick from becoming navigation or other commands. In one embodiment,a live-man configuration may be required for any navigation of theremote mobile workstation (4) at all, to confirm with an additionallevel of certainty that navigation commands, and therefore movement ofthe remote mobile workstation (4), are appropriate. In anotherembodiment, the system may be configured such that when one input deviceis providing inputs to navigate, and a second device then starts toprovide inputs to navigate, the system switches from a state wherein itis navigating based upon navigation commands received from the firstdevice to a state wherein it is navigating based upon navigationcommands received from the second device. Such a configuration may bedesired, for example, in a scenario wherein a pilot is operating with awireless mouse having a battery that suddenly dies; the operator canswitch to navigating with the keyboard or other input device andnavigation will continue using such keyboard or other input devicewithout regard to the mouse with dead battery.

Referring to FIG. 2N, with the left main wheel removed, the wheelmounting feature (60) on a bracket coupled to the movable front castersuspension arm (52) is visible, as is the axis of rotation (64) of thefront caster suspension arm (52) relative to the main mobile base frame(76) or “battery tray assembly”. We have found that having the frontcaster capable of rotating upward relative to the main mobile base frame(76), as shown in FIG. 2R, assists with retaining stability of theoverall remote mobile workstation (4) when the remote mobile workstation(4) encounters a slope, such as an ADA-approved wheelchair ramp, or afloor obstacle, such as an electrical cord or step in the floor surface.The mass of the remote mobile workstation (4) biases all five wheels tostay on the ground, but when the front or rear wheels of the remotemobile workstation (4) encounter a slope or ramp, until the entire setof wheels is approximately co-planar on the slope or ramp, the frontwheel and associated suspension arm (52) are free to rotate up relativeto the main mobile base frame (76), increasing the odds of keeping allfive wheels in contact with the ground surface, and thereby increasingstability as well as controllability (the control system may beconfigured to prefer that the driven wheels, in the depicted embodimentthe wheels 40, remain in contact with the ground surface). When anobstacle, such as a power cord or uneven floor, is encountered duringmovement/navigation of the mobile base (34), the rotating action of thefront wheel and associated suspension arm (52) again prevent dramaticpitching of the entire mobile base (34) assembly by rotating up and outof the way to accommodate the obstacle as the mobile base (34) navigatesover the obstacle. Referring back to FIG. 2O, two range-of-motionlimiting cushion pads (62) enforce a rotational range of motion upon thesuspension arm (52) and associated front caster (50). With the outerbumper skin removed from the front bumper, the underlying bumper cushion(44), made from a compliant material such as polyurethane in anenergy-absorbing geometry such as a waffle geometry as shown, isvisible. A similar construct underlies the rear bumper skin (46), asshown in FIG. 2P (element 70). In another embodiment, one or more of thewheels or casters may be coupled to the main frame of the remote mobileworkstation using a damped suspension to assist with damping and/orshock absorption. Gyro**

FIG. 2P shows the battery cover assembly removed to reveal theunderlying battery (68), which preferably is a 12 volt DC automotivestyle battery, such as the 51 amp-hour battery sold under the tradenameDominator® by Deka, Inc. A rigid bumper substructure (72) is showncoupled to the main frame, or “battery tray assembly”, (76) of themobile base (34). Referring to FIG. 2Q, with the rigid substructure (72)removed, the front caster (52) is more visible. The rear rigid bumpersubstructure (74) is visible in FIG. 2Q with the rear bumper cushion(70) removed. FIG. 2R depicts a main frame (76) rotatably (78) coupledto the front caster suspension arm. FIG. 2S shows an undersideorthogonal view of a similar assembly to illustrate the positions of thefive wheels relative to each other. Again the two main wheels (4—onlyone of the two shown in FIG. 2S) are in differential drive formation,with active electromechanical drive capability through brushlesshubmotors integrated into these wheels (4). The front caster (50) andrear casters (56) are free to roll along the floor surface, and also torotate about the caster axis of rotation.

Referring to FIGS. 2T to 2Z-12 , various aspects of the head assembly(36) are shown in sequential stages of disassembly. FIG. 2T illustratesspine members (26) still coupled to the head (36). The head assembly(36) comprises a main camera (20), a downward oriented navigation camera(22), a forward-oriented speaker (24) covered by a speaker grill (80), atop front microphone cover grill (82), and a top rear microphone grill(84), shown in the rear orthogonal view of FIG. 2U. A rear housing (86)is fitted with a removable lid (88) to cover a compartment (90), asshown in FIG. 2V, which may be utilized to house portions of one or morewireless communication devices, such as a WiFi radio and/or accesspoint, and/or a cellular modem, such as those featuring 3G or 4G LTEwireless connectivity technology. Indeed, the remote mobile workstation(4) may feature one or many wireless communication devices orconfigurations (92), as described, for example, in U.S. PatentApplication No. 61/621,414 for “System for wireless connectivitycontinuity and quality”, which is incorporated by reference herein inits entirety. FIG. 2W shows a view with one more rear panel removed toreveal part of the main backpanel (102) and main computing board (104)below.

Referring to FIG. 2X, with the grills removed, microphone mountingassemblies (96, 98) are visible, along with the loudspeaker (24) below.The main display (100), preferably a relatively low-energy flat paneldisplay with resolution of at least 640×480 (“480p”) and an aspect ratioof 4:3, such as those available from Samsung Corp. or South Korea, issurrounded by a perimeter housing element (94). Referring to FIG. 2Y,with the display removed, the front side of the main backpanel (102) isvisible; the main backpanel preferably comprises a relatively rigidstructure. FIGS. 2Z and 2Z-1 illustrate slightly different views of theassembly of FIG. 2Y. FIGS. 2Z-2 and 2Z-3 illustrate back-side views of asimilar assembly, the version in FIG. 2Z-3 without the rear head housing(86) in place, thereby revealing the main computing board (104),preferably featuring a relatively high-performance microprocessor suchas one of those sold under the tradename Core® by Intel Corporation, anda sound processing board (106) capable of suppressing echo and noise, asdescribed further below. Referring to FIGS. 2Z-4 and 2Z-5 , right (96)and left (98) microphone mounting assemblies are depicted, whereinbutton-style microphones may be press fitted into the small microphonetongs (110) coupled to each of the depicted suspended masses (96, 98) todampen noise and vibration which otherwise may be passed by the remotemobile workstation (4) to the microphones (not shown here to assist inillustrating the geometry of the tongs and suspensions). The depictedembodiment features five front-oriented cardioid button microphones, andone rear-oriented cardioid button microphone; suitable buttonmicrophones are available from vendors such as MWM or PrimoMic of Japan.Referring to FIGS. 2Z-6 and 2Z-7 , the microphones preferably arecoupled to the microphone tongs (110), which comprise extensions of thesuspended microphone mounting assemblies (96, 98); these assembliespreferably are coupled to four structural posts (108) by fourdamping/coupling structures, such as four figure-8 geometrySantoprene®—from Exxon Corporation) damping elements twisted in anapproximately 45 degree mounting configuration as shown; the posts (108)are coupled to a base plate (112), which may be coupled to the headstructure. We have found such a configuration to provide desirablemicrophone damping qualities given the mobility scenario presentedherein (i.e., isolation from vibrations that are related to systemmobility and system vibrations). Another embodiment may feature a totalof eight microphones, with two rear-facing and six forward facing, toenhance sound detection and management. FIGS. 2Z-8 to 2 z-11 featurefurther states of disassembly, illustrating the underlying speakerenclosure (114), RF antenna (116), and power inverter (118) to assistwith the monitor component power supply.

Referring to FIGS. 3A-3I, various aspects of audio subsystem embodimentsare illustrated. Referring to FIG. 3A, a head (36) comprises 5 forwardfacing microphones (120) and one rear-facing (120) microphone, alongwith a forward facing speaker (24). Looking at one microphone forillustrative simplicity, an analog signal comes from the microphone(128), is converted to digital (130), and goes through simple algebraicsubtraction with an echo model (140) based upon signals that are beingoutput to the speaker (24) in the outgoing loop from the pilotworkstation (142). Error that remains when no signal is incoming intothe microphone may be called residual error (134), and may be dealt withusing a residual echo adaptation algorithm, resulting in relativelyecho-free microphone signal (136) from the subject microphone which maybe further processed. Audio componentry such as that depicted in FIGS.3A-3H may be implemented as part of the hardware and firemote mobileworkstation are the comprises a sound board, such as that describedabove in the head component of the depicted remote mobile workstationand described as element 106; suitable sound boards are available fromsources such as Signal Essence of Mountain View, Calif.

Referring to FIG. 3B, with each microphone undergoing echo cancellationon the sound board (106) as described above (148), directional signalmanagement (150) may be conducted to help “focus” on particular sourcesof sound before encoding and transport to a pilot station (152). Thepilot station may also facilitate a pilot's adjustment of certainparameters for the directional signal management (151), such as aspatial zone or frequency of focus, as described below.

Referring to FIG. 3C, in one exemplary scenario, three forward soundsources (S1-S4) are creating sound wave fronts (154, 156, 158, 160),along with a rearward source (S6) creating a sound wave front (162).Using time of flight mathematics and geometry, the approximate locationof the source may be calculated given the timing of arrival of each wavefront with each of the microphones (120) on the front panel, whichtogether essentially comprise a linear array of microphones (120).Referring to the comparison chart display (164) FIG. 3D, signalscaptured by the rear microphone, plotted in the bottom chart (168) maybe utilized to eliminate remnants of such signals that are detected inthe front microphones, plotted in the top chart (166). Greater frequencyseparation between the signals to the rear and the signals to the frontassists with the noise mitigation; for example, if a very low-frequencysource is rearward and most of the sound sources from the front arerelatively high frequency, then separation and filtering can berelatively successful. The dashed plots show remnants of the rearsignals in the front microphone input, and remnants of the front signalsin the rear microphone input. Various weights may be applied to theresult of the rearward microphone, but too much such weighting mayresult in extracting too much signal from the front—thus somebleedthrough generally is acceptable.

Referring to FIG. 3E, with technologies such as RADAR, what is referredto as the “narrow band assumption” may assist in focusing a “beam” (172)close to a particular source, such as S3, to enable focused detection.Human voice style acoustics, on the other hand, is relatively widebandand multi-octave, which makes narrow beam focusing more difficult.Notwithstanding this, small differences in emphasis may be significantto a pilot operating the remote device and listening from differentlocation, and referring to FIG. 3F, predetermined sectors (labeled inthe illustration as Sectors 1-7) may be created through selection ofmicrophone hardware and array beam focusing techniques to emphasizesound sources in the various sectors. The pilot user interface mayfeature an overlay of the sector reticle to allow the pilot to select,such as with a mouse pointing device, one or more sectors of desiredfocus; generally with rejection of as much noise as possible from therear-oriented microphone. Such selection parameters may be fed into theprocess, as briefly described below in reference to FIG. 3G (element151). In another embodiment, it may be desirable for the pilot to do theequivalent of hearing what is going on in back of his remote mobileworkstation head, in which case it may be desirable to emphasize thesignal coming from the rearward microphone and attempt to rejectbleedthrough from the front microphones. One aspect of the pilot userinterface may allow for a selectable listening from the rearwarddirection mode.

In another embodiment wherein drivetrain or other downward-located noisemay be an issue, the system may feature a downward-oriented microphoneto enable subtraction of this undesired noise.

In another embodiment, what is known as “head-related transform”algorithms may be utilized to present a pilot with simulated vectored orthree-dimensional sound presentation, so that the remote pilot can wearheadphones or have a particular loudspeaker setup at his pilotworkstation and be presented with sound that seems to be 3-dimensionalin its directionality, movement, etc, akin to the presentation oftenprovided in a modern movie theater or “surround sound” configuration.

Referring to FIG. 3G, in another embodiment, source localization may beautomated with source location searching (174) algorithms that output anestimated location (176) for a given source, along with a confidence(178) in that location. Such algorithms may be known as “group forcesearch” algorithms, which may, for example, be configured to look inevery possible direction and use wavefront timing across the variousmicrophones to predict vectoring and/or positioning of the sound source.These may be input into a mapping module (180) followed by a spatialfilter in the time domain (182) to output optimal (184) and noisereference (186) signals into a spatial filter module functioning in thetime domain (188) and outputting to an automatic gain control (190), tobe encoded for network transport to the pilot (152).

Referring to FIG. 3H, on the output side at the remote mobileworkstation, decoded signals from the pilot workstation (192), coming inthrough peer to peer connectivity with the pilot workstation, may gothrough automatic gain control (194), dynamic high pass filtering (196),volume control (198), transmit path gain control (200), conversion toanalog, and transmission to the loudspeaker (24) in the remote mobileworkstation head (36) for broadcast to the locals around the remotemobile workstation (4). Various parameters in this chain may be remotelyadjustable by the pilot (i.e., through software controls in the userinterface of the pilot software), such as volume, to enable the pilot tospeak at a relatively constant volume, but to select various modes ofoutput on the local side, such as a “whisper” mode or “public speaking”mode. Referring to FIG. 3I, software algorithms such as those availablefrom Nuance, Inc., or Dragon, Inc., may be utilized at various stages ofthe audio system (preferably as close to the microphone input aspossible to avoid errors or losses before the conversion) to provideconversion of spoken word to text. Further, given textual naturallanguage, one or more of the various software-based translation enginesmay be utilized to translate to almost any language before reproductionon the other end using closed captioning (say scrolled across the bottomof a display) or voice simulation; indeed, the voice simulation stylealso may be selected to simulate a particular accent, gender, etc. FIG.3I shows an embodiment wherein an English-speaking (204) pilot (8)converses with a Spanish-speaking (202) local via voice simulationoutputs at both speakers and/or closed captioning for textual reading oneach end, all facilitated by network connectivity (206) between thelocations.

Referring to FIG. 4 , a login gateway provided by a lobby system (6)prevents someone from easily trying to connect with and operate a remotemobile workstation without appropriate authentication and/orcredentials. In one embodiment, a pilot at his workstation (208) hassome login means to get into the lobby (6) and responds (216) with apassword, private key, or the like. With this challenge met, thecontroller board on the mobile workstation (210) may then make up analphanumeric security code (say “345&*”), and the challenge from it maybe something like, “please sign 345&* using your private key. Theresponse may be the signature itself. Given access to operate the remotemobile workstation, a timeout may be configured to only allow 30 secondsor so of operation before going back to the authentication routine. Sucha configuration facilitates a scenario wherein anyone who hacks into oneof the computer boards of the remote mobile workstation will have noability to make the remote mobile workstation (4, 210, 212) drive,operate the cameras, etc—because the system always relies on the lobby(6) to be in the loop. In other words, the lobby may be configured toallow the motor board that it is ok to drive and operate based upon theauthentication; this may be important in allowing only authenticatedusers to drive the system; the trusted base in such configuration is themotor board, not the entire remote computing system, which may enhancesecurity in the event that someone is able to hack into any board on thesystem other than the motor board.

Referring to FIGS. 5A and 5B, with a camera mounted to the perimeter ofa monitor or display (100, 124), the gaze orientation (126) of thedepicted pilot or local may give the counterparty the feeling that thedepicted pilot or local is not providing direct eye contact. In otherwords, the gaze of the viewer to the display (124, 100) is not directlyaimed at the nearby intaking camera (122, 20). Referring to FIGS. 5C and5D, this may be mitigated by placing the cameras (122, 20) in geometricconfigurations wherein they capture through pinholes formed into thedisplays themselves, at locations selected to substantially align with atypical eye position of a person being shown on the display (124, 100).

Referring to FIGS. 6A-6G, various aspects of certain embodiments pilotworkstation and pilot user interface configuration are illustrated.Referring to FIG. 6A, a typical pilot workstation (208) may comprise alaptop computing system (10), which may have an internal or externalcamera (222) associated with it, such as an external model c910 cameraby Logitech, Inc. An external microphone/speaker device with noise andecho cancelling may also be useful, such as the Chat60® by ClearOne,Inc. Login to pilot software may present a setup dialog box (226) asshown in FIG. 6B. After setup is confirmed, preferably using a lobbysystem as an intermediary, as described above, the operational pilotuser interface (227) may appear as in FIG. 6C, with a main navigationalscreen (228) from the main camera on the remote mobile workstation, anavigational camera view (230) from the navigational camera on theremote mobile workstation, a pilot self-view (232), a pilot-controlledoutgoing status message (234), and controls for microphone level (246),sound level (248), setup configuration (240), session pause (242),session termination (244), remote mobile workstation battery status(236), and connectivity with remote mobile workstation status (238). Inone embodiment a portion of a circular arc is overlaid over each of thelarge camera views (252, 250) to indicate that if the pilot is to mouseclick onto one of these graphical overlays, the remote mobileworkstation will follow such a course. Such overlays are calculatedusing the known turning geometry of the mobile base and the vectoring bythe master input device (mouse, joystick, etc). In other words, a vectoris projected out of the particular camera to a point where the pilot'smouse has been placed in the user interface; then a linear and angularvelocity may be calculated to match the particular circular arc andnavigate the remote mobile workstation where the mouse is affirmativelyclicked (the user interface may be configured to turn the activeoverlaid arc to a different color or brightness when it is beingutilized to navigate the remote mobile workstation). Thus to move aroundan obstacle, the pilot need only orient the graphical overlay around theobstacle, and engage the forward drive by clicking with a mouse,advancing with a joystick or other device, or steering/advancing with akeypad; to move in reverse, the same process may be utilized, and anappropriate circular arc may be overlaid onto the user interface in thereverse direction (mainly viewable in the lower navigational camera).FIGS. 6D-6G show screens representative of connectivity check buttonengagement (254—a small window appears overlaid onto the user interfacefeaturing network communications related information such as incomingbandwidth, outgoing bandwidth, ping time, incoming packet loss, outgoingpacket loss, and general connectivity signal strength to the pertinentwireless access point, cell tower, etc.), moving the mouse to the upperscreen image to get the zoom control (256—to facilitate optical and/ordigital zoom of the pertinent camera), selecting the tools configurationcontrol (226—bringing up the setup/tools dialog box as described abovein reference to login), and placing a session on hold (258) to stop thecameras and microphones on each end while also retaining theconnectivity for selectable re-activation of the session with activecameras and microphones on each end.

Referring to FIG. 7A, a pilot is operating a remote mobile workstation(4) in a conference room setting with locals (30). In one embodiment, inone of the corners of the display, a small window shows video feedbackto the locals regarding what is being captured by the nearby maincamera, so that such locals may have some feedback regarding what isbeing viewed by the remote pilot, how the scene appears to the pilot,etc. FIG. 7B illustrates an embodiment (260) with a longer display tofacilitate split-screen type of presentation, wherein the top screenportion (262) may be utilized to project the pilot's face image, and thelower screen portion to project some other desired image or video, suchas an image of the pilot's screen or one or more graphical userinterface windows presented therein, a document, photo, whiteboard,passenger (as described below), or the like, or videos thereof.

Referring to FIG. 7C, an embodiment features a projector (266)configured to project upon a wall or other object, under the control ofthe remote pilot. A wireless (such as Bluetooth™ enabled) removable USBdongle (268) may be configured to pass video or other images orinformation to a nearby projector, screen, or computing device from theremote mobile workstation (4) at the control of the pilot. One or moretablet computers (270) may be removably housed (272) and wirelesslyconnected to the remote mobile workstation, such as by using the remotemobile workstation as a wireless access point, to provide locals withfurther connectivity and sharing of documents and images/videos with thepilot (in other words, the remote mobile workstation itself may beutilized in a manner akin to being a mobile wireless access pointthrough which the locals may connect tablets to the internet, or simplyconnect tablets to the remote mobile workstation or to the pilot forsharing of information). In one embodiment the tablets may be tetheredto the mobile workstation (4) to prevent them from being taken too faraway from the mobile workstation (4). Referring to FIG. 7D, forillustrative purposes a remote mobile workstation (4) embodiment isequipped with many extra features, including a wheeled (276) trailer(274) that may be towed by a towing hitch (278) to allow for storage ofitems such as in a shopping environment; a basket (280) directly coupledto the remote mobile workstation (4) for additional or optional storage;a lightweight robotic arm with grasper (282) for light manipulation ofitems such as elevator buttons and ADA-compliant doors; a small thermalprinter (290) which may be utilized to print small documents such asreceipts; a credit card point of sale transaction device (288) forprocessing transactions; a Square® credit card point of sale transactiondevice (292) for processing transactions; a near field communicationsdevice (286) for near field communications with enabled credit cards orother devices; a bar code reader (284); an infrared camera (300) forvisualizing heat and in darkness; a relatively high-quality zoom camera(298); a smoke detector (302); a carbon monoxide detector (304). Giventhe mobile platform with onboard power and connectivity to nearbynetworks, many sensors and components may be integrated and/or coupledwith a remote mobile workstation (4). In one embodiment, one or morelaser pointers or projectors, or one or more cameras or imaging devices,may be operatively coupled to the arm to allow the pilot to directprojection or image capture of such devices in the room that is local tothe remote mobile workstation (4). Wireless devices, such as Bluetooth™or RF remote controllers (296) or keyboards/mice (294) may be utilizedby locals to adjust the remote mobile workstation (4) or provide variousinputs. For example, in a conference room environment, it may bedesirable for the locals around a conference table to have a wirelessremote (296) capable of turning up or down the volume, or muting thespeaker and/or microphone, of the remote mobile workstation as itvirtually represents the pilot. Similarly, a wireless remote may beutilized to interact with the remote mobile workstation and/or thepilot. Referring to FIG. 7E, in one embodiment the system may beconfigured to display a QR code (306) for locals to capture andunderstand better a diagnostic challenge or error with the remote mobileworkstation (4); such QR code may be captured by a cellphone camera orsimilar device, and utilized to connect with a networked informationsystem to provide further information to the pertinent local. One ormore wireless microphones (not shown), such as lapel style microphonesmay be removably coupled to the remote mobile workstation (4) to beremoved and placed upon or adjacent to speakers participating in a givendiscussion, to provide relatively direct source isolation and noisereduction. In one embodiment, a pilot may have the functional equivalentof a mixing board in his or her pilot workstation interface, such thatthe pilot may raise or lower the relative volume of various speakers whohave such wireless microphones. In another embodiment, the lightweightrobotic arm (282) may be utilized to hold a microphone toward a selectedsource, in a manner akin to what television reporters are accustomed todoing with interviewees to manage noise and focus on the sound source.In one such embodiment, a microphone may be integrated into thestructure of the robotic arm to be on the ready when needed, butunobtrusive when not needed.

Referring to FIGS. 7F-7H, a combination of a light (308) aligned with astripe feature (310) on the base (2) and/or a stripe feature (316) onthe remote mobile workstation (4), as well as reflective corner markers(312, 314), assist the pilot with parking the remote mobile workstationwhile using the lower navigational camera. In other words, as a pilot isattempting to park a remote mobile workstation (4) onto a charging dock(2) in the orientation required to connect the charging interface of thecharging dock with the charging contacts of the bottom of the remotemobile workstation, it may be helpful for the pilot, likely examiningthe output of the downward-oriented navigational camera (22), to havesome geometric cues as to his state of alignment with the remote mobileworkstation relative to the charging dock. In another embodiment, an“auto parking” feature may assist a pilot in parking correctly on thedock through the selection of a user interface control that preferablyonly is enabled if the pilot has reached a required proximity with acharging dock (which may be confirmed by a near field sensor or othersensing configuration).

Referring to FIGS. 8A-8C, brushless hubmotors, such as those availablefrom Suzhou Bafang Electric Motor Science-Technology Co., LTD of China,may be utilized in the differential drive system of the remote mobileworkstation to provide quiet, inexpensive navigation actuation.Brushless motors typically have three phases (A, B, C), with coils (326,328, 330) in between the phases and permanent magnets (318, 320, 322) ina ringlike configuration around the perimeter. The motor rotates (324)by changing the magnetic field in the coils and changing the directionof current (332, 334, 336). The motors preferably comprise three Halleffect sensors (338, 340, 342) that may be utilized to detect rotationalorientation within about 60 degrees. 60 out of 360 degrees may be anacceptable factor of navigational precision with a wheel that is about 5inches in diameter operating on a typical carpeted or hard floorsurface, but to further smooth out driving (i.e., to avoid jerkiness andsnakelike patterns where smooth and relatively straight or arcuatedriving is preferred between two points), the 2 axis gyro and 3 axisaccelerometer of the board in the mobile base may be utilized in thecontrol loop. The gyro and accelerometer may be operated atapproximately 2 kilohertz, to have high update rates for smoothingcontrol features. One of the unique aspects of this relativelyhigh-torque hub-motor drive system is that it does not featurehigh-precision optical encoders as may be typical in certain otherelectromechanical systems such as robots. Thus the blending of somemotor position feedback from the Hall effector sensors, as well asgeneral navigational dynamic feedback from the gyro and accelerometer,may be desirable in retaining a smoothness of driving, to the benefit ofnot only the remote pilot viewing the world of the locals through one ormore camera mounted to the dynamic vehicle, but also to the locals whogenerally are sitting, walking, and standing around the remote mobileworkstation in the remote location.

Referring to FIG. 8C, a motor shunting circuit (344, 346, 348, 350) maybe configured to shunt the motors with one or more relays when thesystem power is lost, allowing for relatively low motor velocity to pushthe system out of the way, but not high velocities that could lead to arunaway system on a slope (due to the fact that a “short” or shuntedcircuit will allow substantially unlimited current to flow between thephase points out-of-phase, which results in a form of velocity-dependentelectromagnetic braking: the permanent magnets moving past coils willproduce currents that will become a rolling resistance means byproducing torque; thus at relatively low velocity, low current willprovide low electromechanical resistance; if a higher velocity isattempted, there will be more current and more electromechanicalresistance; thus a remote mobile workstation left unpowered on a hill islikely to only slowly move downhill, if at all).

A motor control loop is featured in FIG. 8B. In one embodiment, thebottom layer of the control loop is torque based; at first the torque ofthe motor may be controlled, and this may be accomplished by controllingthe current. Given the 3 phase motor configuration of the abovedescribed embodiment, that means that there are three control loops thatcontrol current through each of the three different phases/coils.Mathematically it sums up to approximately zero because they are allinterconnected. The current that is actually put through the motorwindings is measured, and this facilitates commanding the amount oftorque that is demanded for the particular scenario. Once torque hasbeen controlled, the position of the rotor is desirably controlled aswell. In one embodiment, a super high precision understanding ofrotational motor position is not required; the position can be used tocommutate the motor using a fixed 60-degree control algorithm, whichsome would refer to as a “square wave controller”, as opposed to a “sinewave controller”. In such a configuration, the electrical phase isalways being kept 60 degrees ahead of the position for peak torque andefficiency, and the velocity definitely is of interest, as a pilot of aparticular remote mobile workstation typically may be trying to commandthe remote mobile workstation to “go here”, “now go over there . . .fast”—so it is desirable to have a velocity controller that sits on topof the torque controller and commutation/square wave controller. Thevelocity controller may be configured to take the output of the motor'scommutation (i.e., the above embodiment uses Hall effect sensors), dosome conventional integration and measurement, and determine the angularvelocity of the wheel. The operatively coupled 3-axis accelerometer maybe utilized to measure actual rotational velocity of the remote mobileworkstation as an overall dynamic structure. All of this information maybe examined relative to what the pilot is commanding in the pilotsoftware from the pilot workstation, and in the end, torque commands arecreated. Preferably the torque and velocity controllers areproportional-integral-derivative, or “PID”, controllers. Some tuning ofthe algorithms may be useful in finding smoothness in particular statessuch as standing still with active motors in the aforementioneddifferential drive configuration. For example, if it is desirable to beable to accelerate from zero velocity up to a top speed of between 1 and3 meters/second, it may be useful to have a lot of gain in thecontroller, but then such relatively high gains may result in detectableoscillation of the system (i.e., when it is oscillating due to bouncingby 60 degrees between two phases) as it is sitting in an attempted“still” position, which may be relatively undesirable. Thus in oneembodiment, the controller may be configured to turn down the gains atlower velocities (a “dynamic gain profile” configuration). Given thatthe torque and velocity loops are PID loops, they may be made more orless reactive by changing the P, I, and/or D values, to, for example,turn down the torque to let the drive system be more relaxed. In oneembodiment there may be a performance gain realized by adding afeed-forward term to the PID controller to allow for enhanced control ofvelocity and acceleration within prescribed ramp-up parameters whenslowing down and accelerating the mobile workstation (4).

Referring back to the control loop of FIG. 8B, a commanded velocity(360) is input to a velocity controller (352), which outputs a commandedtorque (362) to the current controller (354). The current controller(354) outputs current (364) to the motor (356) that is measured (366) inthe upper loop. Speed (358) of the motor (356) is computed and fed back(368) to the velocity controller (352).

Using such a control loop, if the pilot is not commanding anything, thenthere is zero velocity command, and with zero velocity commanded, if alocal were to push against the remote mobile workstation to get it tomove out of the way, for example, it would actually push back againstsuch person, because it would be trying to neutralize the velocity beingphysically put into the system; this may not be desirable if anobjective is to allow locals to move the system manually. To addressthis issue, in one embodiment the velocity controller may be basicallyturned off—say after an externally-applied torque passes a given presetthreshold amount, with fallback to doing torque control. If thethreshold is set to zero torque, then an external load could push theremote mobile workstation around with little or no resistance. Thiswould be the equivalent of turning off the controller and having zerocurrent flowing through the motors—which may be nice for energy savings,but may not be good for a remote mobile workstation sitting on a slopedramp, which could lose connectivity and start rolling/movingundesirably. The gyro and accelerometer may be used to mitigate suchissues. For example, it is known that we have 1G of gravity pointinggravity down. Typically we have zero G forward with the remote mobileworkstation sitting still on level ground. If the remote mobileworkstation is on a 30 degree slope (very steep—unrealistic scenario forillustrative purposes) sitting still while it loses connectivity to apilot, it will be sensing a forward acceleration from gravity alone ofabout ½ G. This may be fed back into the torque controller locally, andthus the current may be elevated to keep the remote mobile workstationfrom rolling downhill. Thus a predefined configuration may be createdwherein acceleration is still read while the controller is attempting tohold the system still, and this acceleration may be fed back into thecontroller. So if connectivity is lost, the system may remainsubstantially in place, but if the ½ G threshold is exceeded (say, forexample, a local wants to push the system out of the way while it issitting still on the slope), the local can, indeed, push it out of theway—in a manner similar to pushing it to move it as if it were on levelground—and the local may let go of the system and it will controllablydecelerate back down to zero again to hold its position.

In a “touch control” embodiment, the control system may be configured toonly allow the system to go into freewheeling mode (i.e., wherein it maybe easily pushed by a local) when a local depresses a particularhardware switch or software switch. In one embodiment, a capacitive orresistance switching configuration may allow the system to go intofreewheeling mode only when the touch of a local on a particular portionof the system is detected, such as on a particular pull or pushhandle—and the system may be further configured to return to controlmode and decelerate back to zero velocity once such “touch sensing” isno longer detected.

Referring to FIG. 9 , a typical office scenario map is depicted—it maybe mapped manually by a pilot operating a remote mobile workstation, ormay be mapped by a mapping system configured autonomously explore andmap. A remote mobile workstation or other system may use varioustechniques to gain an understanding of where it is relative to such map,such as by contacting nearby Bluetooth™ LE, Bluetooth™ 4.0, WiFi,infrared, or other beacons which may be placed at various locations andconfigured to transmit map coordinate information. A remote mobileworkstation may also be configured to detect various wireless accesspoints and associate predetermined locations with them, to triangulatebased upon various wireless access point locations, similar to themanner in which cell tower triangulation may be utilized to localize acellphone, but at a smaller and preferably more exacting scale. Variouscharging docks (2), each of which may be configured to have andbroadcast a unique identification name or number, and each of which maybe located at a known location, may serve as indoor localizationbeacons, with technologies such as Bluetooth ™ LE, nearfieldcommunication, WiFi, or infrared data transmission to transmit andtransceive with a local remote mobile workstation. In one embodiment, anautonomous system such as a mapping robot with laser scanning and otherdetecting sensing may be sent into an office environment such as thatdepicted in FIG. 9 to map such environment and provide such informationto one or more remote mobile workstation systems configured to operatewithin such environment. Simultaneous localization and mapping, or“SLAM”, techniques may be utilized in such mapping configurations.

In one embodiment, with multiple different WiFi adaptor radios and atleast one cellular mobile adaptor, such as a 4G adaptor, operated withina head unit of a particular remote mobile workstation, loss of signalmay be less likely. Further, known geometric differences in location oftwo or more adaptors physically coupled to the same remote mobileworkstation may be utilized for time-of-flight based localization of theremote mobile workstation.

In one embodiment, one particular remote mobile workstation withrelatively high levels of connectivity (such as multiple 4G mobilewireless connections, multiple WiFi adaptors at multiple frequencieswith relatively high-exposure external antennae, etc) may be utilized toseek a remote mobile workstation that has become “stuck” due to loss ofconnectivity, and to function as a mobile wireless access point local tothe stuck remote mobile workstation, such that pilot-based control ofthe stuck remote mobile workstation may be regained, and in oneembodiment, so that the stuck remote mobile workstation may be navigatedout of the local wireless networking “hole” and back into a location ofgood connectivity with infrastructure other than the local connectivityto the rescuing remote mobile workstation.

Referring to FIG. 10 , a peer to peer network is set up between thepilot and the remote mobile workstation after lobby authentication, andaudio and video from the local environment may be passed back to thepilot system using a customized implementation of VP8 (356) which may bedescribed as a configuration that is related to VP8, but that goessignificantly beyond VP8 (indeed, some of the configurations describedbelow are not compatible with VP8). Similarly, pilot commands andaudio/video from the pilot are passed back (358) to the remote mobileworkstation (4). A confirmation loop (354) confirms what is beingreceived at the pilot station (10), and provides related input to theoutgoing packet stream formation directed toward the pilot workstation.Each frame has a timing tag that may be utilized to ensure that “stale”or outdated commands are not obeyed at the remote mobile workstation.

In further detail, in one embodiment, initially a pilot goes through alobby to be authenticated, as described above. If successfully past thelobby and a particular remote mobile workstation is selected by aparticular pilot, a peer to peer channel is opened using a protocol suchas Jingle, which is a well-known extension to the extensible messagingand presence protocol (“XMPP”). Once a peer to peer (or “p2p”) channelis open between a remote mobile workstation and a pilot workstation,video and audio may flow back and forth between the remote mobileworkstation and the pilot workstation using a video/audio compressionformat such as VP8, which is publicly available from Google, Inc. VP8generally is relatively open-loop, so the confirmation loop (354) may beestablished to understand, back at the remote mobile workstation, whichframes have been received by the pilot's computing station, and whichframes have not been received. In one embodiment, we can control at theremote mobile workstation how we encode the frames back at the pilotworkstation, based upon what is actually being received at the pilotworkstation. VP8 has standard compression configurations and keeps oneor two frames in the buffer/cache that can be utilized for furtherprocessing, but in one embodiment, that is inadequate. For example, inone embodiment, video transmission is started with an all-gray coloredimage (in between white and black for each pixel); the the first imagemay be built completely from imported data, which is relatively“expensive” in terms of processing and time. Subsequent to this initialbuild, the system preferably is configured to only bring in changes or“delta” to the subsequent frames thereafter. In one embodiment, videotransmission between the pilot workstation and remote mobile workstationis at about 30 frames per second, which is a lot of data moving back andforth. In one embodiment, a frames going back to about 10 seconds backare stored so that intermediary frames may be essentially created byaveraging, interpolating, and the like. For example, in one embodiment,the system is configured to save frames 2, 4, 8, 16, 32, 64, 128, and256 back from the current frame at issue, this being a relativelyefficient way to provide data back about 10 seconds in a 30 frames persecond environment. If there is a problem with a current frame, ratherthan just dropping the connectivity between pilot and remote mobileworkstation, the system may be configured to first go back into thememory and search for a pertinent nearby frame to present. If such frameis not available, the system may be configured to return to the all grayframe and rebuild forward form there, or interpolate or average betweenknown good frames stored in memory. In one embodiment, when frames aresent using VP8, each is assigned a time stamp or time tag. At thereceiving location (such as the pilot workstation), that tag getstransferred to the outgoing command tags to enable the remote mobileworkstation to see not only the commands, but also what time stamp thecommands are based upon—which enables a control paradigm wherein themore “stale” or old a command is (as determined by the time stamp ortag), the less it is to be obeyed at the remote mobile workstation. Sothe system may be configured to avoid control by outdated commands, anda monitoring or confirmation loop (354) is in place to report back tothe remote mobile workstation how the frame receiving is going at thepilot workstation—if this receiving is not going well, preferably thesystem is configured to have mitigating measures in place to avoidsimply terminating the session between the pilot and remote mobileworkstation.

For example, the confirmation loop (354) may tell the remote mobileworkstation (4) that the pilot workstation did receive frames 4 and 6,but missed frame 5. In such case, the system may be configured to createa substitute frame at the space that was to be occupied by the missingframe 5 (by interpolation, averaging, motion vector analysis, etc; forexample: in one embodiment the system may be configured to take thereceived frame 4 and add part of the calculated motion vector betweenframes 4 and 6). The substitute frame may not be perfect, but it may bedesirable relative to having a pause or gap between frames, which maydisrupt the flow of communication between a connected pilot and locals.The substitute frame may be called 5′ or “5 prime” so that it is labeledas a non-actual frame that was created to fill a gap. In one embodiment,the fact that the pilot computing system software just created a “prime”in the frame 5 slot will be reported back in the confirmation loop (354)to the remote mobile workstation (4). If the pilot workstation does notreceive frames 6, 7, or 8 either, the pilot workstation will be prettyfar out into potential inaccuracy in terms of creating primes for thosespots—and at some point may be configured to decide that it is guessingtoo much (say at the position of frame 9), and may be configured to justreturn to a re-presentation of a somewhat stale frame 4, the last realframe in possession of the pilot workstation. In one embodiment, ifframe 5 did not show up as expected, and so a 6 prime and 7 prime arecreated and presented, and then later the actual frame 5 shows up at thepilot workstation, the system may be configured to catch up as fast aspossible to reality (for example, it may be configured to calculate thevectors and compute an actual frame 6 and actual frame 7, and hope tohave those in time to present actual frame 8 instead of a manufactured8-prime. In one embodiment a “relay server” may be implemented torapidly send lost packets for catch up.

In another embodiment, rather than creating a “prime” frame based upononly one or two previous real frames, multiple frames could bereferenced to identify trends in the movement of aspects of the variousimages, etc., subject to processing and timing constraints.

In one embodiment wherein the implementation is related to VP8, VP8features may be utilized dictate how to encode motion vectors, but nothow to determine them. In one embodiment, a technique called “OpticalFlow” (http://en.wikipedia.org/wiki/Optical_flow) is implemented alongwith VP8, which enables an efficient form of image to image correlation,finding the iterative position of various portions of a frame whichleads to determination of the delta or change between two frames. Inother words, to determine where something moved to from frame to frame,it is useful to find out where the item is located from one frame to thenext. An iterative pattern analyzing subportions of an image (say a16×16 pixel portion for grayscale or 8×8 pixel portion for color) usingslight jog up, down, left, right, and dead center may be utilized todetermine where the subportion of the original frame went in the nextsubsequent frame. Once this has been determined, the motion vector maybe determined.

In one embodiment, to save processing time, a “pyramid” style analysismay be conducted wherein the original image is downsampled intosequentially smaller/lower-resolution images. For example, if theoriginal frame is 640×480 pixels, the system may be configured to make a“pyramid” stack of images that get sequentially lower resolution:320×240, 160×120, 80×60; and initially start doing motion vectoranalysis on the smallest resolution image in each stack of each“pyramid”—to get a rough estimate fast about the motion vector. Once thesystem has the answer from the lowest resolution image, this result maybe utilzed as a first guess to address analysis of the next largerimage. As it turns out, such a configuration saves the system lot of theiteration steps (because in each small iterative process, the systemstarts with a guess—and it's preferable/faster to be able to start withan “educated” guess from the previous round of resolution, which may becomputed relatively rapidly). VP8 may require encoding with the highestresolution image in each stack (generally it requires a motion vectorthat is pertinent to the actual resolution of fixed size blocks of theactual image at issue, such as 16×16 pixels for grayscale).

In one embodiment, image processing may be utilized to recognize theeyes of an imaged face in various frames, and whenever a “prime” frameis being fabricated to address a gap in received frames, an open-eyeview may be selected over a closed-eye view, to prevent the receivingparty from interpreting from the presented stream of images that theperson is blinking in a way that may be distracting to the receivingparty.

Video resolution and frame rate for various devices, such as the pilotcamera, remote mobile workstation main camera, remote mobile workstationdownward-facing navigation camera, and other devices, such as a possibleremote mobile workstation-based projector, may be optimized to allowprioritization of bandwidth to a particular function. For example, if apilot is presenting slides using a projector operatively coupled to aremote mobile workstation, and if the remote mobile workstation isstationary and there is no desire to move it during the presentation,frame rate and/or resolution of the navigation camera on the remotemobile workstation may be decreased or made near zero.

Referring to FIG. 11 , a passenger bridge (366) may be added tofacilitate other remotely located persons in observing and participatingin a discussion. Bridges are available from suppliers such aswww.bluejeans.com, to enable passengers to utilize various videoconferencing systems, such as those available from PolyCom, Inc., CiscoSystems, Inc., and others, to join a conference. One pilot workstation(10) equipped with software specifically configured to interact with aremote mobile workstation must be available for the pilot (8), and in atypical bridge configuration, the “passengers” will be “along for theride” from an audio and video perspective due to the connectivitythrough the bridge, but will not be in control of the remote mobileworkstation without the pilot-side software configuration.

Many specific embodiments may be assembled to address particular needsusing various aspects of the aforementioned embodiments. For example, inone embodiment, a remote mobile workstation may be integrated with localbuilding security to allow motion sensors on an alarm system within thevicinity of an operating remote mobile workstation to be temporarilydisabled (i.e., to not set of the building alarm while the remote mobileworkstation is navigating about indoors as allowed by loginauthentication and privileges), based upon the location of the remotemobile workstation as determined by mapping and/or localization featuresof the system, such as those described above. In another embodiment,control features of the physical environment of the remote mobileworkstation (4), such as elevator buttons, light switches, and dooropener actuation buttons may be integrated with the controls paradigm ofthe pilot workstation through network connectivity configuration suchthat the pilot may operate such functions of the remote environment ashe navigates around such environment with the remote mobile workstation(4). In another embodiment, a system with features such as thosedescribed in reference to FIG. 7D may be used for point of sale commerce(to avoid ever having to wait in a checkout line, a person may simplyaccompany or meet up with such a system). In another embodiment,pertinent advice from an expert (for example, an expert plumber mayassist a customer with merchandise questions in a hardware store) may bedelivered in-store by a remote pilot with such a configuration. Inanother embodiment, firefighters may be allowed to check out a buildingusing smoke detectors, carbon monoxide detectors, video, audio, and thelike, from a location remote to a particular building that may be underfire alarm. In one embodiment, certain authorities or personnel may begiven circumstantial access to one or more (or all) remote mobileworkstations (4) in a particular building or environment. For example,in the event of a fire or burglar alarm, local fire and/or policeauthorities may be given instant access to all operational mobileworkstations (4) at the location of the sounding alarm so that certainfire and/or law enforcement personnel may inspect the area through pilotworkstations as other members of their staff get in vehicles andapproach more conventionally.

Various exemplary embodiments of the invention are described herein.Reference is made to these examples in a non-limiting sense. They areprovided to illustrate more broadly applicable aspects of the invention.Various changes may be made to the invention described and equivalentsmay be substituted without departing from the true spirit and scope ofthe invention. In addition, many modifications may be made to adapt aparticular situation, material, composition of matter, process, processact(s) or step(s) to the objective(s), spirit or scope of the presentinvention. Further, as will be appreciated by those with skill in theart that each of the individual variations described and illustratedherein has discrete components and features which may be readilyseparated from or combined with the features of any of the other severalembodiments without departing from the scope or spirit of the presentinventions. All such modifications are intended to be within the scopeof claims associated with this disclosure.

The invention includes methods that may be performed using the subjectdevices and/or systems. The methods may comprise the act of providingsuch a suitable device. Such provision may be performed by the end user.In other words, the “providing” act merely requires the end user obtain,access, approach, position, set-up, activate, power-up or otherwise actto provide the requisite device in the subject method. Methods recitedherein may be carried out in any order of the recited events which islogically possible, as well as in the recited order of events.

Exemplary aspects of the invention, together with details regardingmaterial selection and manufacture have been set forth above. As forother details of the present invention, these may be appreciated inconnection with the above-referenced patents and publications as well asgenerally known or appreciated by those with skill in the art. The samemay hold true with respect to method-based aspects of the invention interms of additional acts as commonly or logically employed.

In addition, though the invention has been described in reference toseveral examples optionally incorporating various features, theinvention is not to be limited to that which is described or indicatedas contemplated with respect to each variation of the invention. Variouschanges may be made to the invention described and equivalents (whetherrecited herein or not included for the sake of some brevity) may besubstituted without departing from the true spirit and scope of theinvention. In addition, where a range of values is provided, it isunderstood that every intervening value, between the upper and lowerlimit of that range and any other stated or intervening value in thatstated range, is encompassed within the invention.

Also, it is contemplated that any optional feature of the inventivevariations described may be set forth and claimed independently, or incombination with any one or more of the features described herein.Reference to a singular item, includes the possibility that there areplural of the same items present. More specifically, as used herein andin claims associated hereto, the singular forms “a,” “an,” “said,” and“the” include plural referents unless the specifically stated otherwise.In other words, use of the articles allow for “at least one” of thesubject item in the description above as well as claims associated withthis disclosure. It is further noted that such claims may be drafted toexclude any optional element. As such, this statement is intended toserve as antecedent basis for use of such exclusive terminology as“solely,” “only” and the like in connection with the recitation of claimelements, or use of a “negative” limitation.

Without the use of such exclusive terminology, the term “comprising” inclaims associated with this disclosure shall allow for the inclusion ofany additional element—irrespective of whether a given number ofelements are enumerated in such claims, or the addition of a featurecould be regarded as transforming the nature of an element set forth insuch claims. Except as specifically defined herein, all technical andscientific terms used herein are to be given as broad a commonlyunderstood meaning as possible while maintaining claim validity.

The breadth of the present invention is not to be limited to theexamples provided and/or the subject specification, but rather only bythe scope of claim language associated with this disclosure.

The invention claimed is:
 1. A remote presence system, comprising: aremote mobile workstation comprising a controller and a drive system;and a pilot workstation operatively coupled to the remote mobileworkstation in a peer-to-peer network configuration, the pilotworkstation comprising a first input device and a second input device,such that navigation commands configured to be input from a pilotoperating on the first input device and the second input device arecommunicated to the remote mobile workstation and utilized to providenavigation control of the drive system by the controller; wherein thecontroller of the remote mobile workstation is configured to switch froma first state of navigation control of the drive system, wherein thefirst state of navigation control is based upon receiving, at a firsttime, a first navigation command generated by and communicated from thefirst input device of the pilot workstation at the remote mobileworkstation, to a second state of navigation control of the drivesystem, wherein switching to the second state of navigation control isbased upon an operation failure of the first input device during thefirst state of navigation control of the drive system, and receiving, ata second time later than the first time, a second navigation commandgenerated by and communicated from the second input device of the pilotworkstation at the remote mobile workstation, and upon switching to thesecond state of navigation control, disregard any subsequent navigationcommands at the remote mobile workstation received after the second timecommunicated from the first input device of the pilot workstation havingthe operational failure.
 2. The remote presence system of claim 1,wherein the controller of the remote mobile workstation is configured torequire a third live-man input device to be activated by the user at thepilot workstation at a same time as the navigation commands input fromthe first input device and the second input device in order for thenavigation commands to be interpreted as drive commands to the drivesystem of the remote mobile workstation.